Operating systems are a well-known component of computer systems. An operating system (or OS) is a software application (or program), or group of software applications, which comprises computer code or instructions which are executed by the processor (or CPU) of a computer system upon which the operating system is run. The operating system manages the hardware resources of the computer system upon which it is run. These hardware resources include, for example, processors, memory, storage and input/output (IO) interfaces. The operating system allows other software applications to be run or executed on the computer system and provides various functionality and services, including interfaces to the hardware resources of the computer system, which support the running of other software applications. An operating system therefore forms a core or central component of a computer system and is responsible for the running of all other applications that are to be executed by the processor(s) of the computer system upon which it runs. As such, the operating system normally forms a trusted component of the computer system, which is also responsible for ensuring the security of the processing environment by preventing applications that are run by the operating system from negatively impacting or interfering with the operation of the hardware resources of the computer system or the other applications that are being run by the computer system. It is commonly a requirement, therefore, that software applications can only interact with the hardware resources of the computer system using the interfaces provided by, and managed by, the operating system.
Examples of well-known operating systems include Microsoft Windows, Mac OS X, Linux, BSD, IBM z/OS, Android and iOS, however, it will be appreciated that there are other operating system that may be used.
A software application comprises computer code or instructions which identify actions that a processor of a computer system is to take when it executes the software application. A software application that is being executed by a processor may be referred to as a task or process. When a software application is executed by a processor (i.e. during the execution of a process), the calculations and operations that are performed result in the generation (and potentially the subsequent alteration) of various data such as the intermediate results of calculations, physical memory addresses where data may be stored and indicators of which instruction is to be processed next. This data provides an indication of the processing state (or context) of the process and is typically stored in registers of the processor while the process is being executed. A process may therefore be considered as comprising a copy (or image) of the program code of the software application that is being executed and a current processing state.
A processor which comprises a single processing core is only capable of processing or executing a single instruction at a time and, as a result, can only execute a single software application at a time. A processor (or a group of processors) may, however, comprise multiple processing cores, in which case the processor can process multiple instructions at a time, with one instruction being processed by each processing core. A processor with multiple processing cores may therefore execute multiple software applications at a time. The concept of executing multiple software applications at the same time is referred to as multi-processing or multi-tasking. Most modern operating systems support multi-processing and manage the hardware resources that each process that is being run on the computer system may access to avoid any conflict or interference between the processes that are currently running. In order to support multi-processing, the operating system maintains information about the processes that are being run on the computer system. The operating system may store this information in special memory structures, which may be referred to as Process Control Blocks (PCBs). The nature of the information that is maintained may vary between different operating systems, however commonly includes information such as an identifier for the process, the hardware resources such as memory and I/O devices that are allocated to the process, as well as other control and security attributes for the process. It will be appreciated that much more information about processes may be maintained by the operating system. Furthermore, it will be appreciated that the operating system may maintain some of the information about the processes in other data structures, such as tables, which may be referenced back to a particular process by an identifier of the process. The state of execution of a process may therefore be considered to be a combination of its processing state and the additional information about the process that is maintained by the operating system.
Whilst multi-processing can be achieved by the presence of multiple processing cores in a computer system, most modern operating systems also enable multi-processing to be achieved (or at least emulated) by a concept known as concurrent execution of processes (or concurrent processing or concurrency). The concurrent execution of processes means that on a single processing core, multiple processes are processed during the same period of time. Concurrency is achieved by allowing the processing core to switch from executing a process to executing a different process, even if the processing of the process hasn't yet finished. This process of switching the process that is currently being processed may be referred to as context-switching. In order for context-switching to take place, it is necessary to save the processing state of the process that is currently being executed, before the context-switch to a different process happens. This means that when the processor switches back to processing that process, the processing state of the process can be reloaded and execution can be resumed from the point where the previous context-switch occurred. The operating system may support concurrency in several ways. The operating system may assist in storing the process state of the processes when they are context-switched out of the processor—this may be stored in the process control blocks. The operating system may also determine when, and in what order, each process is executed and how long each process is executed before the processor context-switches to a different process. This component of the operating system may be referred to as a scheduler. There are well-known strategies for determining when context-switching should take place, examples of such strategies include co-operative strategies, pre-emptive strategies and time-sharing strategies, however it will be appreciated that any such strategies may be used.
In addition to the multi-processing and concurrent execution of multiple processes by a computer system, various methods are also known which allow for a single process to contain multiple streams or threads of execution (which may be referred to simply as threads), such that different portions of code of the process can be executed at the same time or concurrently. This concept may also be referred to as multi-threading. Multi-threading means that different execution paths within a process can be followed at the same time, by executing instructions at the same time on different processing cores, or during the same time period, by executing the threads concurrently on the same processing core. In this respect, threads are similar to processes. However, whereas processes have their own allocated resources, threads share the resources of the process to which they belong. The operating system may support multi-threading, however, since threads share the processes to which they belong, it is also possible for multi-threading behaviour to be implemented (or at least emulated) by the program code of a particular application, without requiring the support of the operating system.
It is a well-known technique in software application development to utilise multiple co-operating processes and or threads in order to improve the operation of software applications.
The operating system may make information about the processes and/or threads that are running on a computer system available to other applications or a user of the computer system. The operating system may also provide an interface to other applications or the user of the computer system which allows interaction with the processes that are running on the computer system. As examples, this interface may allow the identification, diagnosis (i.e. as to whether the process is malfunctioning), suspension or termination of a particular process. It will be appreciated that the may be other ways in which interaction is enabled with the processes that are running on a computer system.
Software debugging tools, commonly referred to as software debuggers or just debuggers, are well known tools that may be used to investigate the operation of another software application. Software debugging tools are themselves software applications that interact with the operating system and/or hardware, such as the processor, of the operating system to control the execution of a running process executing a different software application and to observe the instructions that are executed by that process as well as the data structures that form the processing state of that process at various points during that process's execution. In this manner debuggers enable the operation of a process to be investigated and the functionality of the software application to be derived. Debuggers may also provide additional functionality which aid the user of the debugger in understanding the process or software application that is being debugged. As examples, the software debugger may include other tools such as a disassembler or a decompiler to display the code in a higher level language such as source code that may be easier for the user to understand. Debuggers are commonly used by programmers as part of their normal application development and/or maintenance activities, such as, for example, investigating and fixing errors or bugs in a software application.
In general, debuggers work by setting breakpoints in the execution of a software application. These breakpoints result in the execution of the application being stopped at the breakpoint and control being passed to the debugger. The debugger is then able to observe the program code or instructions that were being run at that breakpoint as well as the current state of the application's data structures.
In practice, the operation of a debugger may be supported by functionality provided by various subsystems with a computer system. Supporting functionality for debugging may, for example, be provided at the hardware level, for example by the CPU, which may provide special debug registers, tracing support and interrupt instructions that are designed to support debugging. Similarly, debugging functionality may, for example, be embedded within the operating system. Different debugging tools may make use of different elements of debugging support functionality provided by the different layers of the computer system and may therefore operate in a variety of different ways.
When a program (or software) is being executed by a processor, the environment in which the execution is being performed is a so-called “white-box” environment if the user (or a third party) has access to the processing so that the user can observe and alter the execution of the program (e.g. by running a suitable debugger).
When a software application is run in a white-box environment, a third party or attacker who does not have access to the original source code for the software application may attempt to use debugging tools to deduce information about the operation of a software application. This process may also be referred to as reverse-engineering of the software application. An attacker may use the knowledge obtained from reverse-engineering to meet their own goals, such as, for example, identifying attack points for an exploit, bypassing protection measures to allow unauthorised distribution of the application or modifying the application to include malicious code. There are a number of debugging tools which have been created with sophisticated automated facilities to aid in such reverse-engineering attacks.
Various techniques are known which aim to make it more difficult to debug a software application. These techniques may be referred to as anti-debugging techniques. As discussed above, there are a range of different techniques that may be used to debug a software application by utilising the various debugging support functionality provided by different layers or parts of the computer system. Each of these debugging techniques may result in artefacts or side effects occurring within the processing environment which may be detected. The different debugging techniques may also have their own respective weaknesses, in so far as they may be unable to handle or monitor particular operations that occur within a process. Generally, methods of anti-debugging rely on exploiting these side effects or weaknesses to detect or disable the operation of a debugger. The following are examples of the types of anti-debugging techniques that may be employed to protect a software application from being debugged:                Hardware and register based anti-debugging involves monitoring the data stored in the registers of the CPU itself. As mentioned above, some CPU's provide support for debugging by way of specific registers that can be used to allow breakpoints in the execution of an application. It is possible therefore to detect the operation of a debugging tool by monitoring the status of these registers. If the operation of a debugging tool is detected, the application may take an appropriate action to frustrate the debugging attempt.        Application Programming Interface (API) based anti-debugging involves calling various API methods of the Operating System, which can reveal that a debugger is operating on an application. Some of these API methods may be provided by the operating system with the express purpose of allowing an application to determine whether it is being debugged. Others of the API methods may be affected by the operation of a debugger, such that the use of a debugger can be determined by checking the operation of these methods, even though the purpose for including these API methods in the operating system may not have been expressly for the purpose of allowing an application to determine whether it is being debugged. Once the operation of a debugger is detected, the application may then take an appropriate action to frustrate the debugging attempt.        Process and thread blocking detection anti-debugging involves directly checking the process and/or thread management information maintained by the operating system.        Modified code anti-debugging is based on the principle that some debuggers insert breakpoints into an application by modifying the code of the application to include special instructions that cause the processor to pass control of the execution back to the debugger. It is possible to detect the inclusion of these breakpoints in the executed code by carrying out an Integrity Verification (IV) check, such as, for example, a cyclic redundancy check (CRC), to check whether the current image of the application that is being executed corresponds to an image of the application that hasn't been modified (i.e. to include breakpoints). By carrying out this self inspection or integrity verification, the application can detect whether it is being run under the control of a debugger and take an appropriate action to frustrate the debugging attempt.        Exception-based anti-debugging is based on the principle that certain exceptions that might be raised during the execution of a software application are handled differently when the application is being run under the control of a debugger than when the application is run normally. As an example, some debuggers may handle certain types of exceptions, such that the corresponding exception handling code in the application is never called when the application is run under the control of a debugger, yet would have been called were the application run normally. Conversely, other exceptions may be handled by the application when it is run normally, but are not correctly handled when run under a debugger, thereby causing the application to crash or terminate when it is run under the control of a debugger. Again, the operation of a debugger can be detected by the use of such exceptions and the application may then take an appropriate action to frustrate the debugging attempt.        Timing and latency based anti-debugging involves checking the timings at various points in the execution of the application to determine whether there is significant latency or delay between the two points of the execution. If there is a significant delay, then this code by indicative of the execution of the application being interrupted through the operation of a debugging tool. Again, having detected the operation of a debugging tool, the application may take an appropriate action to frustrate the debugging attempt.        
It will be appreciated that other types of anti-debugging techniques may be used. Furthermore, it will be appreciated that the above-described anti-debugging techniques are broad categories of types of techniques that may be used and that within each of these categories there may be many different implementations of the type of anti-debugging technique, each of which may be tailored to the nuances of an individual debugger or operating system.